Skip to content

Add cel-expr-python backend#485

Open
anuraaga wants to merge 14 commits into
bufbuild:mainfrom
anuraaga:cel-python
Open

Add cel-expr-python backend#485
anuraaga wants to merge 14 commits into
bufbuild:mainfrom
anuraaga:cel-python

Conversation

@anuraaga

@anuraaga anuraaga commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

This adds an implementation using cel-expr-python. It is much faster, orders of magnitude, but has many gotchas. This is currently a rough draft PR to show the concept and list up what it means

  • We will need to keep the celpy fallback due to limited platform support for cel-expr-python. The engine glue code is large and duplicated by necessity
    • No musl wheels
    • No free-threaded Python support
    • Build requires Bazel so no sdist. Normally extensions like protobuf-py-ext can still be used with the relatively simple "install rust" prereq on less common platforms, but here it's impossible
  • We can't add an automatic dependency on it using markers so it is always opt-in by user
    • No marker for musl vs glibc
    • No marker for free-threaded Python
  • We have to copy protobuf-py messages to google.protobuf messages. There is some perf impact but we can consider it minimal. But our core engine becomes google.protobuf, even though we want our ecosystem to be built on protobuf-py
  • Our own code (i.e. extra_func.py) gets more brittle inheriting google.protobuf's weak type annotations. No docstrings on the API we use

@anuraaga
anuraaga marked this pull request as draft July 3, 2026 05:32
@anuraaga
anuraaga marked this pull request as ready for review July 7, 2026 13:00
@anuraaga
anuraaga requested a review from ajeetdsouza July 7, 2026 13:00
@anuraaga

anuraaga commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

While I have some reservations, we have decided to go with this approach - we'll see if issues come up going forward. To make it easier to rethink, I have kept the implementations largely separate - anyways, the CEL APIs being quite different necessitates a lot of the general control flow duplication, and it would get quite surgical to try to extract common logic bits. We can reconsider in the future if things settle

Comment on lines +24 to +34
# protobuf 7+ removed FieldDescriptor.label / LABEL_REPEATED in favour of is_repeated.
_FieldDescriptorClass = google_descriptor.FieldDescriptor
if hasattr(_FieldDescriptorClass, "is_repeated"):

def _is_repeated(field: google_descriptor.FieldDescriptor) -> bool:
return field.is_repeated

else:

def _is_repeated(field: google_descriptor.FieldDescriptor) -> bool:
return field.label == google_descriptor.FieldDescriptor.LABEL_REPEATED

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're doing the same thing in rules.py - can we just import it from there?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah extracted to a util

@@ -0,0 +1,1686 @@
# Copyright 2023-2026 Buf Technologies, Inc.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this file appears to be duplicated from internal/extra_func.py.

@anuraaga anuraaga Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - at first I thought there was at least a need for structural duplication since the cel types are different for the two libraries, and thought sharing logic would get complicated then. I realized a simple argument wrapper on the celpy side avoids both duplication and surgery, just a couple of functions left that are not matches between the two

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants